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ABSTRACT 



A pilot study was conducted to evaluate the instructional 
effectiveness of the SIGNATURES program (Harcourt Brace) , an elementary 
reading program that aimed to reflect the best thinking about reading 
instruction. In developing the program, the publishers reviewed current 
curriculum guides and content standards for most states and from professional 
groups such as the National Council of Teachers of English and the 
International Reading Association. Also, they met in small focus groups with 
himdreds of teachers and consulted with several nationally recognized reading 
educators regarding content and instructional practices. A total of 18 
teachers from Illinois, Massachusetts, Minnesota, Missouri, New Jersey, and 
Ohio participated in the study: 9 at grade 1, and 9 at grade 4. The teachers 
had not used the program previously. A quasi -experimental pretest-posttest 
design was used. At grade 1 the mean Skills Test score increased from 78.5% 
before instruction to 86.1% after instruction; at grade 4, the mean Skills 
Test score increased from 72.5% to 83.1%. Both groups showed a significant 
gain in the Stanford 9 Total Reading score, and additional analysis was 
conducted to see which subtests may have contributed to those gains. Findings 
suggest that SIGNATURES is instructionally effective, and this appears to 
hold up across grade levels. Contains 4 tables of data. (NKA) 
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A Study of the Instructional Effectiveness 
of the SIGNATURES Program 



The purpose of this report is to describe a pilot study that was conducted to evaluate the 
instructional effectiveness of the SIGNATURES program. 

Background Information 

Developing an instructional program is a complex and demanding task that takes many 
months, if not years. In order to create an elementary reading program that reflected the 
best thinking about reading instruction, Harcourt Brace took several factors into 
consideration. First, they reviewed current curriculum guides and content standards for 
most states and many large school districts to identify common instructional elements and 
their developmental placement. As part of this process, they examined the standards 
emanating from professional groups such as the National Council of Teachers of English 
and the International Reading Association to determine how the new program could meet 
those standards. Second, they met in small focus groups with hundreds of teachers 
around the country to find out what they desired in a reading program, and which features 
would be most beneficial to them in their day-to-day teaching. Finally, they consulted 
with several nationally recognized reading educators for their advice regarding content 
and instructional practices. 

In spite of the considerable effort that publishers expend in creating new programs, they 
are increasingly being asked to provide evidence that the programs they develop are 
effective. “Do students really learn if they use this program?” has become the rallying cry 
for some policy makers and state departments of education. Most educators would agree 
that no program is “teacher proof’ and capable of producing student gains in spite of the 
teacher. Rather, research has consistently shown that the classroom teacher is the single, 
most important variable in the instructional equation. None the less, good teachers desire 
effective programs to help them achieve their goals. 

It is within this context that Harcourt Brace Educational Publishers asked the Center for 
Innovation in Assessment at Indiana University to conduct an independent pilot study to 
evaluate the effectiveness of SIGNATURES. Harcourt Brace Educational Publishers 
sought out volunteer teachers to participate in the study; the Department of Test Services 
assisted in designing, securing, and scoring the data collection instruments; and Indiana 
University conducted the study and analyzed the data that were collected. 

Research Questions 

The following research questions guided the design, procedures, and data analysis of the 
pilot study: 
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Is SIGNATURES instructionally effective? Do selected themes significantly 
increase students’ understanding of key reading skills, concepts, and strategies as 
measured by a program Skills Test? As measured by a standardized achievement 
test? 

Does SIGNATURES have a positive impact on students’ attitudes and interests? 
Do selected themes significantly increase students ’ attitudes toward reading as 
measured by an informal survey? 

Design and Procedures of the Study 

Because of cost and time considerations, the study was limited to two grades - grades 1, 
and 4. It was believed that these two grades would give a good cross-sectional look at the 
primary and upper elementary levels of the program. 

Teachers who volunteered to participate in the study had not used the program previously. 
The teachers were allowed to select one theme to pilot over a four- to six-week period. 
They also agreed to administer data collection instruments before beginning instruction 
and again after completing instruction. 

A total of 18 teachers volunteered to participate in the study: 9 at grade 1, and 9 at grade 
4. The participating teachers came from six states: Illinois, Massachusetts, Minnesota, 
Missouri, New Jersey, and Ohio. 

A quasi-experimental pretest-posttest design was used. Before instruction began, 
students were administered the following pretests: (1) the appropriate SIGNATURES 
Skills Test; (2) the reading section of the Stanford 9 Achievement Test, Ninth Edition 
(Abbreviated Battery); and (3) and the Elementary Reading Attitude Survey, an informal 
instrument designed to measure attitudes toward reading. 

Following the pretests, the teachers taught the selected themes using the resources and 
procedures contained in SIGNATURES. Teachers also received training from 
experienced consultants in methods of implementing and using the program. 

Instructional time varied from four to six weeks depending on the themes taught. In most 
cases, SIGNATURES replaced the existing reading program. 

Upon completion of the theme, students were administered the same three instruments as 
posttests. All data collection instruments were returned to Indiana University where they 
were processed. The Stanford 9 Achievement Tests were scored at the Harcourt Brace 
Educational Measurement Scoring Center. The SIGNATURES Skills Tests and the 
Elementary Reading Attitude Surveys were scored at Indiana University, and all of the 
data were analyzed at Indiana University. 

Table I summarizes the skills taught in the themes that were piloted by teachers in the 
study. 



TABLE 1 

Content of the SIGNATURES’ Themes 
Taught by the Pilot Teachers 



Theme 


Grade 1 


Grade 4 


1 


Vocabulary 

Keywords 

Decoding 

Initial, medial, final consonants 
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Vocabulary 
Key Words 
Decoding 

Structural analysis: prefixes, suffixes, 
roots 

Literary Appreciation 
Narrative elements 
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Vocabulary 
Key Words 
Comprehension 
Cause and effect 
Sequence 


4 


Vocabulary 


Vocabulary 




Key Words 


Key Words 




Decoding 


Context clues 




Short vowels/long vowels 


Comprehension 




Structural analysis: inflected forms 


Cause and effect 




Comprehension 


Main idea/details 




Make predictions 
Study Skills 
Alphabetical order 


Sequence 
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Vocabulary 


Vocabulary 




Key Words 


Key Words 




Decoding 


Context clues 




Short vowels/long vowels 


Comprehension 




Consonant clusters/digraphs 


Main idea/details 




Structural analysis: inflected forms 


Summarize/paraphrase 




Comprehension 


Sequence 




Main idea/details 


Study Skills 
Reference sources 
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Vocabulary 
Key Words 
Comprehension 
Compare/contrast 
Fact/opinion 
Main idea/details 
Study Skills 
Reference Sources 
Literary Analysis 
Narrative elements 
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As Table 1 indicates, the pilot study included a broad sample of reading skills from the 
strands of decoding, vocabulary, comprehension, literary appreciation, and study skills. 

Findings 



Table 2 summarizes the pre- and posttest means and standard deviations for the Skills 
Test, the Stanford 9, and the Elementary Reading Attitude Survey for both grades. The 
means and standard deviations for the Skills Tests and the Elementary Reading Attitude 
Survey are presented in percentages; those for the Stanford 9 are in raw scores. 

TABLE 2 

Means and Standard Deviations of 
Pretests and Posttests 



Grade 


Skills 

Test 


Stanf 
Total F 


'ord 9 
leading . 


Attitude 

Survey 


Grade 1 (N=194) 


Mean 


SD 


Mean 


SD 


Mean 


SD 


Pretest 


78.5 


20.5 


53.5 


10.8 


76.5 


13.7 


Posttest 


86.1 


16.5 


56.3 


9.4 


72.8 


15.7 


Grade 4 (N= 157) 














Pretest 


72.5 


22.0 


35.8 


9.4 


71.1 


14.8 


Posttest 


83.1 


16.1 


38.2 


8.6 


70.5 


17.1 “1 



At grade 1, the mean Skills Test score increased from 78.5% before instruction to 86.1% 
after instruction. This increase was highly significant (t=9.417; pK.OOOl). Likewise, the 
raw score means for Total Reading on the Stanford 9 increased from 53.5 to 56.3 as a 
result of instruction (t=5.858; pK.OOOl). The raw score means were converted to scale 
scores, percentiles, and grade equivalents in order to interpret the increase in normative 
scores. The mean scale score increased from 548 to 558; the percentile scores for the 
means increased from 65 to 72; and the grade equivalent scores for the means increased 
from 2.2 to 2.3. Unlike the achievement measures, the mean score on the Attitude Survey 
decreased from 76.5% to 72.8% percent. This decline was significant (t=4.085' 

PK.OOOl). 

At grade 4, the mean Skills Test score increased from 72.5% to 83.1%, a statistically 
significant gain (t=8.368; pK.OOOl). The Total Reading raw score mean on the Stanford 
9 increased from 35.8 to 38.2. This increase is also significant (t=6.573; pK.OOOl). 

When interpreted in terms of normative scores, the scale score went from 638 to 646, the 
percentile score from 50 to 58, and the grade equivalent score from 4.8 to 5.3. The mean 
score for the Attitude Survey decreased from 71.1% to 70.5%. This decline was not 
significant (t=.642; pK.522). 

Because both groups showed a significant gain in the Stanford 9 Total Reading score, an 
additional analysis was conducted to see which subtests may have contributed to those 
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gains. Table 3 shows the pre- and posttest subtest means and standard deviations for the 
Stanford 9 for grade 1. 



TABLE 3 

Means and Standard Deviations of the 
Stanford 9 Subtests for Grade 1 
(N=194) 



Test 


Word 

Study Skills 


Word 

Reading 


Reading 

Comprehension 




Mean 


SD 


Mean 


SD 


Mean 


SD 


Pretest 


15.2 


3.4 


14.9 


4.4 


22.6 


4.9 


Posttest 


16.3 


3.3 


15.6 


3.5 


24.0 


4.3 



As Table 3 shows, mean scores increased for all three subtests. The increases for Word 
Study Skills {t=5.240; p<.0001) and for Reading Comprehension (t=5.169; p<.0001) 
were highly significant. However, the increase in Word Reading was moderately 
significant {t=2.3; p<.025). 

A similar analysis was conducted for grade 4. At grade 4, the Stanford 9 yields two 
subtest scores - Vocabulary and Reading Comprehension. Table 4 presents the pre- and 
posttest means and standard deviations for these subtests. 



TABLE 4 

Means and Standard Deviations of the 
Stanford 9 Subtests for Grade 4 
(N=157) 



Test 


Vocabulary 


Reading 

Comprehension 




Mean 


SD 


Mean 


SD 


Pretest 


15.3 


3.4 


20.1 


6.9 


Posttest 


16.2 


3.2 


21.8 


6.1 



As in grade 1, mean scores increased on each of the subtests. The increase in Vocabulary 
(r=5.7; p<.0001) and the increase in Reading Comprehension {t=5.4; pK.OOOl) were 
both highly significant. 
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Conclusions 



The following conclusions can be made based on the findings: 

• SIGNATURES is instructional ly effective. Students using SIGNATURES increase 
their understanding of key reading skills, concepts, and strategies. 

• The instructional effectiveness of SIGNATURES appears to hold up across grade 
levels. In this study, students in grades 1 and 4 showed similar increases in reading 
achievement. 

• The instructional effectiveness of SIGNATURES also appears to demonstrate itself 
on both criterion-referenced measures related to the program as well as standardized 
achievement tests that are independent of the program. 

• At grade 1, the average increase on the Stanford 9 was approximately 7 percentile 
points after completing instruction in only one theme, with the greatest gains coming 
in Word Study. 

• At grade 4, the average increase on the Stanford 9 was about 11 percentile points after 
completing instruction in only one theme, with the greatest gains coming in Reading 
Comprehension. 

• There were decreases in student attitudes toward reading. At grade 1 the decrease 
was statistically significant; at grade 4 the decline was not significant. The decrease in 
student attitudes was an unexpected finding of the study. Since it is likely that 
attitudes and interests toward reading are formed over a longer period of time, it is 
difficult to explain what may have contributed to these decreases. 
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